在过去的几年中，神经网络（NN）从实验室环境中发展为许多现实世界中的最新问题。结果表明，NN模型（即它们的重量和偏见）在训练过程中的重量空间中的独特轨迹上演变。随后，这种神经网络模型（称为模型动物园）的人群将在体重空间中形成结构。我们认为，这些结构的几何形状，曲率和平滑度包含有关训练状态的信息，并且可以揭示单个模型的潜在特性。使用这种模型动物园，可以研究（i）模型分析的新方法，（ii）发现未知的学习动力学，（iii）学习此类人群的丰富表示形式，或（iv）利用模型动物园来用于NN权重和NN权重的生成模型偏见。不幸的是，缺乏标准化模型动物园和可用的基准可以显着增加摩擦，以进一步研究NNS人群。通过这项工作，我们发布了一个新颖的模型动物园数据集，其中包含系统生成和多样化的NN模型种群，以进行进一步研究。总共提出的模型动物园数据集基于八个图像数据集，由27个模型动物园组成，该模型动物园训练有不同的超参数组合，包括50'360唯一的NN型号以及其稀疏双胞胎，导致超过3'844'360收集的型号。 。此外，对于模型动物园数据，我们提供了对动物园的深入分析，并为多个下游任务提供了基准。该数据集可在www.modelzoos.cc上找到。

给定模型动物园的神经网络权重的学习表示是一个新兴而具有挑战性的领域，从模型检查到神经体系结构搜索或知识蒸馏，具有许多潜在的应用。最近，在模型动物园进行训练的自动编码器能够学习一个超代理，该代表体捕获了动物园中模型的内在和外在特性。在这项工作中，我们扩展了超代表，以供生成使用以采样新的模型权重。我们提出的是层损失归一化，我们证明，这是基于超代表拓扑生成高性能模型和几种采样方法的关键。使用我们的方法生成的模型是多种多样的，性能的，并且能够超过强大的基准，从而在下游任务上进行了评估：初始化，合奏采样和传递学习。我们的结果表明，通过超代理通过过度代理，知识聚集从模型动物园到新模型的潜力，从而为新的研究方向铺平了途径。

给定模型动物园的神经网络权重的学习表示是一个新兴而具有挑战性的领域，从模型检查到神经体系结构搜索或知识蒸馏，具有许多潜在的应用。最近，在模型动物园进行训练的自动编码器能够学习一个超代理，该代表体捕获了动物园中模型的内在和外在特性。在这项工作中，我们扩展了超代表性的生成用途，以品尝新的模型权重作为预训练。我们提出的是层损失归一化，我们证明，这是生成高性能模型和基于超代表经验密度的采样方法的关键。使用我们的方法生成的模型是多种多样的，性能的，并且能够超过传统基线的转移学习。我们的结果表明，通过超代理通过过度代理，知识聚集从模型动物园到新模型的潜力，从而为新的研究方向铺平了途径。

训练大型神经网络可以通过训练较小的超网络来预测大型参数。最近发布的图形Hypernetwork（GHN）对100万个较小的ImageNet体系结构进行了这种训练，能够预测大型未见网络（例如Resnet-50）的参数。尽管具有预测参数的网络在源任务上失去了性能，但已发现预测参数可用于对其他任务进行微调。我们研究了基于同一GHN的微调是否仍然对GHN经过培训后出版的新型强架构仍然有用。我们发现，对于诸如Convnext之类的最新体系结构，GHN初始化比Resnet-50有用。一个潜在的原因是，新型体系结构从用于训练GHN的建筑的分布转移增加。我们还发现，预测参数缺乏成功调整梯度下降的微调参数所需的多样性。我们通过将简单的后处理技术应用于预测参数，然后再对目标任务进行微调并改善Resnet-50和Convnext的微调，从而缓解了这一限制。

This article concerns Bayesian inference using deep linear networks with output dimension one. In the interpolating (zero noise) regime we show that with Gaussian weight priors and MSE negative log-likelihood loss both the predictive posterior and the Bayesian model evidence can be written in closed form in terms of a class of meromorphic special functions called Meijer-G functions. These results are non-asymptotic and hold for any training dataset, network depth, and hidden layer widths, giving exact solutions to Bayesian interpolation using a deep Gaussian process with a Euclidean covariance at each layer. Through novel asymptotic expansions of Meijer-G functions, a rich new picture of the role of depth emerges. Specifically, we find that the posteriors in deep linear networks with data-independent priors are the same as in shallow networks with evidence maximizing data-dependent priors. In this sense, deep linear networks make provably optimal predictions. We also prove that, starting from data-agnostic priors, Bayesian model evidence in wide networks is only maximized at infinite depth. This gives a principled reason to prefer deeper networks (at least in the linear case). Finally, our results show that with data-agnostic priors a novel notion of effective depth given by \[\#\text{hidden layers}\times\frac{\#\text{training data}}{\text{network width}}\] determines the Bayesian posterior in wide linear networks, giving rigorous new scaling laws for generalization error.

Vision-based tactile sensors have gained extensive attention in the robotics community. The sensors are highly expected to be capable of extracting contact information i.e. haptic information during in-hand manipulation. This nature of tactile sensors makes them a perfect match for haptic feedback applications. In this paper, we propose a contact force estimation method using the vision-based tactile sensor DIGIT, and apply it to a position-force teleoperation architecture for force feedback. The force estimation is done by building a depth map for DIGIT gel surface deformation measurement and applying a regression algorithm on estimated depth data and ground truth force data to get the depth-force relationship. The experiment is performed by constructing a grasping force feedback system with a haptic device as a leader robot and a parallel robot gripper as a follower robot, where the DIGIT sensor is attached to the tip of the robot gripper to estimate the contact force. The preliminary results show the capability of using the low-cost vision-based sensor for force feedback applications.

Deploying machine learning models in production may allow adversaries to infer sensitive information about training data. There is a vast literature analyzing different types of inference risks, ranging from membership inference to reconstruction attacks. Inspired by the success of games (i.e., probabilistic experiments) to study security properties in cryptography, some authors describe privacy inference risks in machine learning using a similar game-based style. However, adversary capabilities and goals are often stated in subtly different ways from one presentation to the other, which makes it hard to relate and compose results. In this paper, we present a game-based framework to systematize the body of knowledge on privacy inference risks in machine learning.

This paper presents a class of new fast non-trainable entropy-based confidence estimation methods for automatic speech recognition. We show how per-frame entropy values can be normalized and aggregated to obtain a confidence measure per unit and per word for Connectionist Temporal Classification (CTC) and Recurrent Neural Network Transducer (RNN-T) models. Proposed methods have similar computational complexity to the traditional method based on the maximum per-frame probability, but they are more adjustable, have a wider effective threshold range, and better push apart the confidence distributions of correct and incorrect words. We evaluate the proposed confidence measures on LibriSpeech test sets, and show that they are up to 2 and 4 times better than confidence estimation based on the maximum per-frame probability at detecting incorrect words for Conformer-CTC and Conformer-RNN-T models, respectively.

Training a neural network requires choosing a suitable learning rate, involving a trade-off between speed and effectiveness of convergence. While there has been considerable theoretical and empirical analysis of how large the learning rate can be, most prior work focuses only on late-stage training. In this work, we introduce the maximal initial learning rate $\eta^{\ast}$ - the largest learning rate at which a randomly initialized neural network can successfully begin training and achieve (at least) a given threshold accuracy. Using a simple approach to estimate $\eta^{\ast}$, we observe that in constant-width fully-connected ReLU networks, $\eta^{\ast}$ demonstrates different behavior to the maximum learning rate later in training. Specifically, we find that $\eta^{\ast}$ is well predicted as a power of $(\text{depth} \times \text{width})$, provided that (i) the width of the network is sufficiently large compared to the depth, and (ii) the input layer of the network is trained at a relatively small learning rate. We further analyze the relationship between $\eta^{\ast}$ and the sharpness $\lambda_{1}$ of the network at initialization, indicating that they are closely though not inversely related. We formally prove bounds for $\lambda_{1}$ in terms of $(\text{depth} \times \text{width})$ that align with our empirical results.

Deep Learning (DL) models tend to perform poorly when the data comes from a distribution different from the training one. In critical applications such as medical imaging, out-of-distribution (OOD) detection helps to identify such data samples, increasing the model's reliability. Recent works have developed DL-based OOD detection that achieves promising results on 2D medical images. However, scaling most of these approaches on 3D images is computationally intractable. Furthermore, the current 3D solutions struggle to achieve acceptable results in detecting even synthetic OOD samples. Such limited performance might indicate that DL often inefficiently embeds large volumetric images. We argue that using the intensity histogram of the original CT or MRI scan as embedding is descriptive enough to run OOD detection. Therefore, we propose a histogram-based method that requires no DL and achieves almost perfect results in this domain. Our proposal is supported two-fold. We evaluate the performance on the publicly available datasets, where our method scores 1.0 AUROC in most setups. And we score second in the Medical Out-of-Distribution challenge without fine-tuning and exploiting task-specific knowledge. Carefully discussing the limitations, we conclude that our method solves the sample-level OOD detection on 3D medical images in the current setting.